Understanding and manipulating face images in-the-wild is of great interest to the computer vision and graphics community, and as a result, has been extensively studied in previous work. Example techniques range from relighting portraits (for example, Y. Wang, Z. Liu, G. Hua, Z. Wen, Z. Zhang, and D. Samaras, “Face Re-lighting from a Single Image Under Harsh Lighting Conditions”, Pages 1-8, June 2007), editing or exaggerating expressions (for example, F. Yang, J. Wang, E. Shechtman, L. Bourdev, and D. Metaxas, Expression Flow for 3d-aware Face Component Transfer, ACM Transactions on Graphics (TOG), volume 30, page 60, ACM, 2011), and even driving facial performances (for example, F. Yang, J. Wang, E. Shechtman, L. Bourdev, and D. Metaxas, “Expression Flow for 3d-aware Face Component Transfer”, ACM Transactions on Graphics (TOG), volume 30, page 60, ACM, 2011). Many of these methods start by explicitly reconstructing facial attributes such as geometry, texture, and illumination, and then editing these attributes in the image. However, reconstructing these attributes is a challenging and often ill-posed task. Previous techniques attempt to address these challenges by either utilizing more data (for example, RGBD video streams) or imposing a strong prior on the reconstruction that is adapted to the particular editing task that is to be solved (for example, utilizing low dimensional geometry). As a result, these techniques tend to be both costly (with respect to use of computational resources) and error-prone. Moreover, such techniques fail to generalize at scale.